-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add SearchAfterMixin for ES search_after capability #4536
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
January 9, 2025 09:44
2262eb3
to
a70ebb7
Compare
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
4 times, most recently
from
January 13, 2025 20:30
2b92b48
to
d93f032
Compare
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
January 15, 2025 20:39
d93f032
to
7a8f3b7
Compare
Ali-D-Akbar
commented
Jan 15, 2025
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
2 times, most recently
from
January 16, 2025 17:19
de8f6a3
to
e2052c4
Compare
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
3 times, most recently
from
January 20, 2025 07:57
66e7fae
to
c3c3901
Compare
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
January 23, 2025 13:26
c3c3901
to
89fad11
Compare
hamza-56
reviewed
Jan 28, 2025
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
2 times, most recently
from
January 29, 2025 04:33
0c21348
to
eed171c
Compare
DawoudSheraz
approved these changes
Jan 29, 2025
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
January 29, 2025 07:42
eed171c
to
61f2f95
Compare
hamza-56
approved these changes
Jan 29, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PROD-4233
Adds a new
SearchAfterMixin
to be added in place ofPkSearchableMixin
that allows using search_after. Once this mixin is used, it will bypass the default search limit of 10k by making multiple calls to ES in case we have more than 10k records in an index.Previously, we faced an issue regarding the search limit, resulting in less records to be returned. We increased the
MAX_RESULT_WINDOW
before but a better way is to use search_after capability for an optimal and flexible result.This PR also adds a v2
CatalogQueryContainsViewSet
that fixes the querying mechanism by filtering the items at the time of when we're executing queries on ES. In v1, ourCatalogQueryContainsViewSet
first searched all the records AND THEN filtered it.Testing Instructions:
update_index
locally in Discovery shell./api/v1/catalog/query_contains/
endpoint and add a sample query like this: http://localhost:18381/api/v2/catalog/query_contains/?course_uuids=2de67490-f748-4efd-8532-b445f7ecc6f9,f9f1e100-668a-4fd5-a966-a127de1f69de&query=org:edXYou can set
ELASTICSEARCH_DSL_QUERYSET_PAGINATION
to your specific value in order to test the behavior. While the end result will be the same but this can affect the number of times the search_after mechanism is called.